Skip to content

refactor(silo): do not overly rely on std::ostream when writing to http buffer - #1079

Merged
taepper merged 2 commits into
mainfrom
better-ndjson-sink
Dec 4, 2025
Merged

refactor(silo): do not overly rely on std::ostream when writing to http buffer#1079
taepper merged 2 commits into
mainfrom
better-ndjson-sink

Conversation

@taepper

@taepper taepper commented Dec 2, 2025

Copy link
Copy Markdown
Collaborator

This changes the way we write to an std::ostream, to guarantee that we do not write to it in too big chunks of data

@taepper
taepper marked this pull request as ready for review December 2, 2025 14:55
@github-actions

github-actions Bot commented Dec 2, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

This is a preview of the changelog of the next release. If this branch is not up-to-date with the current main branch, the changelog may not be accurate. Rebase your branch on the main branch to get the most accurate changelog.

Note that this might contain changes that are on main, but not yet released.

Changelog:

0.9.2 (2025-12-04)

Features

  • benchmarking: add DEBUG custom variable for running debug builds (541afc8)
  • benchmarking: update api-query, enable CSV log file w/ checksum comparisons (3d135d0)
  • build: tag images with branch name again (670817c)
  • documentation: add documentation for sequence storage format (47e4081)
  • silo: add efficient way of computing vertical coverage bitmaps and use them to adapt reference positions (8e7e9f4)
  • silo: add reordering of predicates to apply most selective predicates first (1db8cf0)
  • silo: better compression for sequences (9eb69c7)
  • silo: change IsInCoveredRegion to predicate instead of being its own filter-operator (6fef558)
  • silo: enable rewrite of SymbolInSet expressions under Or (5489117)
  • silo: split compile function into separate rewrite step (27b7e7c)

Bug Fixes

  • build: correctly fix the cppstd for installing the arrow dependency (0c4aed3)
  • build: fix Makefile to not repeatedly invoke conan install (346e6a9)
  • silo: do not add empty coverage bitmaps (5bd32c7)

pflanze
pflanze approved these changes Dec 2, 2025
View reviewed changes

@pflanze pflanze left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess all is good, mentioned doubts in the comments.

Comment thread src/silo/query_engine/exec_node/ndjson_sink.cpp Outdated
Comment thread src/silo/query_engine/exec_node/ndjson_sink.cpp
@pflanze

pflanze commented Dec 2, 2025

Copy link
Copy Markdown
Contributor

(Also, the PR title should refer to std::ostream, not istream, right?)

@taepper taepper changed the title refactor(silo): do not overly rely on std::istream when writing to http buffer refactor(silo): do not overly rely on std::ostream when writing to http buffer Dec 2, 2025
@taepper
taepper force-pushed the better-ndjson-sink branch from ef71a43 to a72bd2c Compare December 3, 2025 10:19
@taepper
taepper force-pushed the better-ndjson-sink branch from a72bd2c to c06fe84 Compare December 4, 2025 09:47
@taepper
taepper merged commit 9f54887 into main Dec 4, 2025
10 checks passed
@taepper
taepper deleted the better-ndjson-sink branch December 4, 2025 10:03
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fengelniederhammer fengelniederhammer Awaiting requested review from fengelniederhammer
1 more reviewer
@pflanze pflanze pflanze approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@taepper @pflanze